Image reconstruction method and device

ABSTRACT

Embodiments of the present application provides an image reconstruction method and device. The method comprises: when a long-focus image set acquisition condition is met, driving the pan-tilt platform of a binocular long-focus and short-focus pan-tilt camera to rotate throughout a field-of-view range, obtaining one long-focus image every preset horizontal and/or vertical angle, and constituting a first long-focus image set with all long-focus images acquired throughout the field-of-view range; receiving a zoom request, and performing interpolating on the first short-focus image currently acquired to obtain a second short-focus image that meets the zoom request; downsampling each long-focus image in the first long-focus image set to obtain a second long-focus image set; dividing the second short-focus image into blocks, searching, for each of the divided blocks, all long-focus images in the second long-focus image set for a matching block; fusing each of the divided blocks in the second short-focus image with the matching block for this divided block to obtain a reconstructed short-focus image. The embodiments of the present application not only retain the advantage of a large field-of-view of the short-focus image, but also improves the definition.

The present application claims the priority of a Chinese patentapplication No. 202010649902.3 filed with the China NationalIntellectual Property Administration on Jul. 8, 2020 and entitled “ImageReconstruction Method and Device”, which is incorporated herein byreference in its entirety.

TECHNICAL FIELD

The present application relates to the technical field of imageprocessing, in particular to an image reconstruction method and device.

BACKGROUND

Due to a large field-of-view angle of a short-focus camera and a smallfield-of-view angle of a long-focus camera, the definition of along-focus image is high and the definition of a short-focus image islow when the images are acquired with an image sensor of the same size.In order to fuse the images with two focal lengths to achieve a zoomingeffect, a software method is needed to improve the spatial resolution ofthe image, that is, the super-resolution algorithm.

Super-resolution refers to the reconstruction of low-resolution imagesinto corresponding high-resolution images. The existing super-resolutionalgorithm generally includes a single-frame super-resolution algorithmand a super-resolution algorithm using high-definition reference images.Single-frame super-resolution algorithm has limited effects on imageenhancement and generally has high algorithm complexity.

The super-resolution method using high-definition reference images takesfull advantages of reference images and images to be processed to obtainhigh-quality images through synthetization. In the existing solution,long-focus and short-focus dual-focus cameras are used to acquire boththe large-field wide-angle image and the high-definition long-focusimage, then the high-resolution information of the long-focus image ismigrated to the repairable area of the short-focus image using theoverlapping area of the fields of view of the wide-angle and long-focuscameras, and the digital zoom is performed based on the repairedshort-focus image. This method is suitable for scenarios where theoverlapping area of an short-focus image and a long-focus image issimilar to other areas. However, if there are significant differencesbetween different parts of the image, the enhancement through localinformation is not significant or even causes errors, resulting in worsepicture effect.

SUMMARY

Embodiments of the present application provide an image reconstructionmethod and device to improve the definition of the short-focus imagewhile preserving the advantages of the large field-of-view of theshort-focus image acquired by the binocular long-focus and short-focuspan-tilt camera.

The technical solution of the embodiment of the present application isimplemented as follows.

An image reconstruction method is provided, including:

when a preset long-focus image set acquisition condition is met, drivinga pan-tilt platform of a binocular long-focus and short-focus pan-tiltcamera to rotate throughout a field-of-view range, obtaining onelong-focus image every preset horizontal and/or vertical angle, andconstituting a first long-focus image set with all long-focus imagesacquired throughout the field-of-view range, wherein adjacent long-focusimages overlap partially;

receiving a zoom request, and performing interpolating on a firstshort-focus image currently acquired by the binocular long-focus andshort-focus pan-tilt camera to obtain a second short-focus image thatmeets the zoom request;

downsampling each long-focus image in the first long-focus image set toobtain a second long-focus image set, wherein a target in eachlong-focus image in the second long-focus image set and the same targetin the second short-focus image are matched in size;

dividing the second short-focus image into blocks, and searching, foreach of the divided blocks, all long-focus images in the secondlong-focus image set for a matching block;

fusing each of the divided blocks in the second short-focus image withthe matching block for this divided block to obtain a reconstructedshort-focus image;

wherein the binocular long-focus and short-focus pan-tilt camera iscomposed of a short-focus camera and a long-focus camera, relativepositions of the short-focus camera and the long-focus camera are fixed,and the short-focus camera and the long-focus camera rotatesimultaneously with the pan-tilt platform.

Wherein, fusing each of the divided blocks in the second short-focusimage with the matching block for this divided block includes:

extracting a high-frequency component of the matching block for each ofthe divided blocks, and fusing each of the divided blocks with thehigh-frequency component of the matching block for this divided block.

Wherein, fusing each of the divided blocks in the second short-focusimage with the high-frequency component of the matching block for thisdivided block includes:

assigning a fusion weight for each matching block according to aprinciple that the larger a similarity between the matching block andits corresponding divided block is, the larger the fusion weight of thematching block is;

multiplying the high-frequency component of each matching block by thecorresponding fusion weight to obtain a result and then superimposingthe result with the corresponding divided block, to obtain thereconstructed short-focus image.

Wherein, assigning the fusion weight for each matching block accordingto the principle that the larger a similarity between the matching blockand its corresponding divided block is, the larger the fusion weight ofthe matching block is includes:

calculating

${W_{k} = ( \frac{E_{k}}{E_{\min}} )^{- \alpha}},$

wherein W_(K) is a fusion weight of a matching block for a kth dividedblock of the second short-focus image,

${E_{k} = \frac{1}{S_{k}}},$

S_(K) is a similarity between the kth divided block of the secondshort-focus image and the matching block,

${E_{\min} = \frac{1}{S_{\max}}},$

S_(max) is a maximum similarity among similarities between all dividedblocks of the second short-focus image and the matching blocks for thedivided blocks, and a is a preset value.

Wherein, searching, for each of the divided blocks, all long-focusimages in the second long-focus image set for the matching blockincludes:

searching, for each of the divided blocks of the second short-focusimage, all the long-focus images in the second long-focus image set fora matching block that matches the divided block in position according toan angle of the pan-tilt platform when the first short-focus image isacquired, an angle of the pan-tilt platform when each long-focus imagein the first long-focus image set is acquired and a field-of-viewdifference between a short-focus lens and a long-focus lens; or

searching, for each of the divided blocks, all the long-focus images inthe second long-focus image set for a matching block having a highestimage similarity with the divided block by using a preset templatematching algorithm.

Wherein, extracting the high-frequency component of the matching blockfor each of the divided blocks in the second short-focus image includes:

downsampling the long-focus image in which the matching block is locatedin the second long-focus image set such that a resolution of thedownsampled image is the same as that of the first short-focus image,then performing interpolating on the downsampled image such that aresolution of the interpolated image is equal to that of the secondshort-focus image, and subtracting a corresponding block of the matchingblock in the interpolated image from the matching block to obtain thehigh-frequency component of the matching block.

Wherein, the preset long-focus image set acquisition condition is met ina case including:

when the binocular long-focus and short-focus pan-tilt camera isstarted, or, when a difference between a photometric value of acurrently acquired short-focus image and a photometric value of along-focus image in a last acquired long-focus image set is greater thana preset first threshold, or when an image similarity between dividedblocks at same positions in the currently acquired short-focus image andin a long-focus image in the last acquired long-focus image set is lessthan a preset second threshold.

Wherein, after the preset long-focus image set acquisition condition ismet and before the pan-tilt platform of the binocular long-focus andshort-focus pan-tilt camera is driven to rotate throughout thefield-of-view range, the method further includes:

determining whether there is no alarm target in a currently acquiredshort-focus image and there is currently no image preview requirement,wherein it is determined that there is no alarm target in the currentlyacquired short-focus image and there is currently no image previewrequirement; and

when it is determined that there is an alarm target in the currentlyacquired short-focus image or there is currently an image previewrequirement, waiting for a preset time, and then returning to the actionof determining whether there is no alarm target in the currentlyacquired short-focus image and there is currently no image previewrequirement.

An image reconstruction device is provided, which includes a memory anda processor that can access the memory having stored thereininstructions which, when executed by the processor, cause the processorto implement steps of any method as described above.

A binocular long-focus and short-focus pan-tilt camera is provided,which includes a short-focus camera, a long-focus camera and a pan-tiltplatform, wherein relative positions of the short-focus camera and thelong-focus camera are fixed, and the short-focus camera and thelong-focus camera rotate simultaneously with the pan-tilt platform; and

each of the short-focus camera and the long-focus camera has aninterface to communicate with the image reconstruction device, and theshort-focus camera and the long-focus camera send an originalshort-focus image and an original long-focus image respectively acquiredby the short-focus camera and the long-focus camera to the imagereconstruction device through the respective communication interfaces.

In the present application, for each of divided blocks in a short-focusimage, a matching block is searched for in a long-focus image, and thenthe matching block is fused with the divided block in a short-focusimage to obtain a reconstructed short-focus image, thereby thereconstructed short-focus image retains the advantages of the largefield-of-view of the short-focus image and has improved definition.

BRIEF DESCRIPTION OF DRAWINGS

In order to more clearly describe the technical solution of theembodiments of the present application and the prior art, drawingsneeded in the embodiments and the prior art will be briefly describedbelow. Obviously, the drawings described below are for only someembodiments of the present application, one of ordinary skills in theart can also obtain other drawings based on the drawings illustratedherein.

FIG. 1 shows the field-of-view relationship between the short-focus lensand the long-focus lens of a binocular long-focus and short-focuspan-tilt camera provided in the embodiment of the present application.

FIG. 2 shows the flow chart of an image reconstruction method providedin an embodiment of the present application.

FIG. 3 shows the flow chart of an image reconstruction method providedin another embodiment of the present application.

FIG. 4 is an example diagram illustrating the relationship between thefield-of-view covered by a short-focus image and the position of apan-tilt platform provided in an embodiment of the present application.

FIG. 5 is a schematic structural diagram of an image reconstructiondevice provided in an embodiment of the present application.

DETAILED DESCRIPTION

In order to make objectives, technical solutions and advantages of thepresent application more apparent, the present application now will bedescribed in detail with reference to the accompanying drawings and byway of examples. Obviously, the embodiments described herein are onlysome of the embodiments of the present application instead of all ofthem. All other embodiments obtained by those skilled in the art basedon the embodiments in the present application fall within the protectionscope of the present application.

The present application is explained in further detail below inconjunction with the drawings and the detailed description.

For ease of understanding, the following explanations are given first:

Focal length: the distance from the center of the lens to the focuswhere light is focused. Generally, more than 6 mm (millimeters) iscalled the long-focus, and less than 6 mm is called the short-focus.

Continuous rotation: it can continuously rotate in a specified directionwithout position limitation.

Image stitching and fusion refers to the merging of two images with acommon part into one image to expand the field-of-view angle and/orimprove the definition. For example, in the monitoring scenario of abinocular long-focus and short-focus pan-tilt camera, according to thecommon part of the long-focus image and the short-focus image, thecommon part in the short-focus image is replaced by the long-focusimage, so as to take both advantages of the large field-of-view angle ofthe short-focus camera and the high definition of the long-focus camera.However, as the picture is enlarged, the low definition problem of thepart from the short-focus image becomes more and more obvious. To solvethis problem, the present application provides the following solutions.

An embodiment of the present application firstly provides a binocularlong-focus and short-focus pan-tilt camera, wherein the pan-tiltplatform can rotate horizontally and vertically. There are two lensesand two sensors. A short-focus sensor and a short-focus lens form ashort-focus camera. The short-focus lens has a larger field-of-view andcan cover a larger monitoring range, while a long-focus sensor and along-focus lens form a long-focus camera. The long-focus lens has asmaller field-of-view but a higher definition.

In one example of the binocular long-focus and short-focus pan-tiltcamera provided by the embodiment of the present application, therelative positions of the short-focus lens and the long-focus lens arefixed, and the optical axes thereof are parallel and as close aspossible to each other. The pan-tilt platform of the binocularlong-focus and short-focus pan-tilt camera can rotate continuouslyhorizontally, or vertically from 0 to 90 degrees, where the horizontaldirection can be parallel to the base plane and the vertical directioncan be perpendicular to the base plane, and the base plane can be anyplane. In this case, the horizontal field-of-view angle of theshort-focus lens is 120 degrees, the vertical field-of-view angle is 68degrees, and the horizontal field-of-view angle of the long-focus lensis 30 degrees, the vertical field-of-view angle is 17 degrees. And thereis a gyroscope to detect the position of the pan-tilt platform. Theresolutions of long-focus and short-focus sensors can be the same ordifferent. However, in order to ensure the resolution of the pictureusing the long-focus lens, that is, to make the resolution of thepicture using the long-focus lens reach a preset resolution threshold,it is recommended to ensure that the resolution of the long-focus sensorreaches or exceeds 1080P, that is, to ensure that the resolution of thepicture using the long-focus lens reaches or exceeds 1080P.

The field-of-view relationship between the short-focus lens and thelong-focus lens remains unchanged in FIG. 1 , that is, the field-of-viewrelationship between the short-focus lens and the long-focus lensremains as shown in FIG. 1 , regardless of how the pan-tilt platformrotates. As shown in FIG. 1 , the area contained in the small rectangleat the center is the field-of-view of the long-focus lens, while thearea contained in the large outside rectangle outside is thefield-of-view of the short-focus lens.

FIG. 2 shows the flow chart of an image reconstruction method providedin an embodiment of the present application. The specific steps are asfollows:

Step 201: when a preset long-focus image set acquisition condition ismet, driving the pan-tilt platform of a binocular long-focus andshort-focus pan-tilt camera to rotate throughout a field-of-view range,obtaining one long-focus image every preset horizontal and/or verticalangle, and constituting a first long-focus image set with all long-focusimages acquired throughout the field-of-view range, wherein adjacentlong-focus images overlap partially.

Step 202: receiving a zoom request, and performing interpolating on afirst short-focus image currently acquired by the binocular long-focusand short-focus pan-tilt camera to obtain a second short-focus imagethat meets the zoom request.

Step 203: downsampling each long-focus image in the first long-focusimage set to obtain a second long-focus image set, wherein a target ineach long-focus image in the second long-focus image set and the sametarget in the second short-focus image are matched in size.

Because the field-of-view of the long-focus lens is smaller than that ofthe short-focus lens, that is, the focal length of the long-focus lensis longer than that of the short-focus lens, according to the principleof optical imaging, the size of a target in the long-focus image islarger than that in the short-focus image. To fuse the long-focus andshort-focus images, the long-focus image in the first long-focus imageset must be reduced in size so that the size of the same target in thelong-focus image is the same as that of the target in the secondshort-focus image.

Step 204: dividing the second short-focus image into blocks, searching,for each of the divided blocks, all long-focus images in the secondlong-focus image set for a matching block.

Step 205: fusing each of the divided blocks in the second short-focusimage with the matching block for this divided block to obtain areconstructed short-focus image.

Wherein the binocular long-focus and short-focus pan-tilt camera iscomposed of a short-focus camera and a long-focus camera, the relativepositions of the short-focus camera and the long-focus camera are fixed,and the short-focus camera and the long-focus camera rotatesimultaneously with the pan-tilt platform.

Through the above mentioned embodiment, for each of the divided blocksin a short-focus image, a matching block is searched for in a long-focusimage, and then the matching block is fused with the divided block in ashort-focus image to obtain a reconstructed short-focus image, therebythe reconstructed short-focus image retains the advantages of the largefield-of-view of the short-focus image and has improved definition.

In an embodiment, in Step 205, fusing each of the divided blocks in thesecond short-focus image with the matching block for this divided blockcan include: extracting the high-frequency component of the matchingblock for each of the divided blocks in the second short-focus image,and fusing each of the divided blocks in the second short-focus imagewith the high-frequency component of the matching block for this dividedblock.

Wherein, the high-frequency component can refer to the image informationin the matching block whose frequency is higher than a preset frequencythreshold. Understandably, the image can be considered as asuperimposition of image information of different frequencies. Thehigher the frequency, the greater the change of image information causedby spatial changes. On the contrary, the lower the frequency, thesmaller the change of image information caused by spatial change.Exemplary, the frequency of image information contained in the solidcolor area of the image is lower, while the frequency of imageinformation contained in the area with complex textures is higher.Therefore, the high-frequency component is often the contour or detailof the image.

Therefore, in the above embodiment, by extracting the high-frequencycomponent of the matching block, the contours or details in thelong-focus image are retained. The high-frequency component of thematching block is fused with the divided block in the second short-focusimage, so that the defect of unclear details in the short-focus imagecan be remedied, and the definition of the reconstructed short-focusimage is improved.

In an embodiment, fusing each of the divided blocks in the secondshort-focus image with the high-frequency component of the matchingblock for this divided block can include: calculating the fusion weightfor each matching block according to the principle that the larger asimilarity between the matching block and its corresponding dividedblock is, the larger the fusion weight of the matching block is;multiplying the high-frequency component of each matching block by thecorresponding fusion weight to obtain a result and then superimposingthe result with the corresponding divided block to obtain thereconstructed short-focus image.

In different application scenarios, the principle that the larger asimilarity between the matching block and its corresponding dividedblock is, the larger the fusion weight of the matching block is canrefer to different principles. At that time, the fusion weight should bepositively related to the similarity. Exemplary, the relationshipbetween the fusion weight and the similarity can be as follows:

W _(k) =aS _(k) +b.

Wherein, W_(k) is the fusion weight of the matching block for the kthdivided block of the second short-focus image, S_(k) is the similaritybetween the kth divided block of the second short-focus image and thematching block, a and b are preset values, and a is greater than 0.

In the above embodiment, when the similarity between the matching blockand its corresponding divided block is higher, the weight assigned tothe matching block is larger, so that when the long-focus image issimilar to the short-focus image, the difference can be eliminated,while when there is a difference between the long-focus image and theshort-focus image, such as when an object enters or leaves, thedifference can be emphasized to avoid reconstruction errors.

In an embodiment, calculating the fusion weight for each matching blockaccording to the principle that the larger a similarity between thematching block and its corresponding divided block is, the larger thefusion weight of the matching block is can include:

calculating

${W_{k} = ( \frac{E_{k}}{E_{\min}} )^{- \alpha}},$

wherein W_(K) is the fusion weight of the matching block for the kthdivided block of the second short-focus image,

${E_{k} = \frac{1}{S_{k}}},$

S_(K) is the similarity between the kth divided block of the secondshort-focus image and the matching block,

${E_{\min} = \frac{1}{S_{\max}}},$

S_(max) is the maximum similarity among the similarities between alldivided blocks of the second short-focus image and the matching blocksfor the divided blocks, and α is a preset value. And a is greater than0. Exemplary, α can be 0.2, 0.5, 1, 1.2 and so on.

In the above embodiment, the calculation manner for determining thefusion weight of the matching block based on the similarity between thedivided block and the matching block is given. Through this manner, thefusion weight calculated can be normalized, which is convenient forsubsequent calculation and processing.

In an embodiment, in Step 204, searching, for each of the dividedblocks, all long-focus images in the second long-focus image set for amatching block can include: searching, for each of the divided blocks ofthe second short-focus image, all the long-focus images in the secondlong-focus image set for a matching block that matches the divided blockin position according to the angle of the pan-tilt platform when thefirst short-focus image is acquired, the angle of the pan-tilt platformwhen each long-focus image in the first long-focus image set isacquired, and the field-of-view difference between the short-focus lensand the long-focus lens; or, searching, for each of the divided blocks,all the long-focus images in the second long-focus image set for amatching block with the highest image similarity with the divided blockby using a preset template matching algorithm.

The matching block with a matched position can refer to that theposition in the real space represented by the matching block matches theposition in the real space represented by the divided block. The spatialposition represented by an image block is the spatial position reflectedby the picture in the image block. It is understandable that in theorythe divided block and matching block corresponding to the divided blockshould be obtained by taking pictures of the same area, so the matchingblock corresponding to the divided block can be determined by searchingfor a matching block that matches the divided block in position.

Because the angle of the pan-tilt platform when the first short-focusimage is acquired, the angle of the pan-tilt platform when eachlong-focus image in the first long-focus image set is acquired, and thefield-of-view difference between the short-focus lens and the long-focuslens can reflect the relative relationship between the spatial positionrepresented by each image block in the first short-focus image and thespatial position represented by each image block in the first long-focusimage, and the second short-focus image and the second long-focus imageare obtained based on the first short-focus image and the firstlong-focus image, respectively, all the long-focus images in the secondlong-focus image set can be searched to obtain a matching block with amatched position according to the angle of the pan-tilt platform whenthe first short-focus image is acquired, the angle of the pan-tiltplatform when each long-focus image in the first long-focus image set isacquired, and the field-of-view difference between the short-focus lensand the long-focus lens.

Furthermore, the preset template matching algorithm can be any templatematching algorithm, including, but not limited to, MAD (Mean AbsoluteDifferences), SAD (Sum of Absolute Differences), SSD (Sum of SquaredDifferences), MSD (Mean Square Differences), and NCC (Normalized CrossCorrelation), etc.

It is understandable that since in theory the divided block and thematching block corresponding to this divided block should be obtained bytaking pictures of the same area, the image similarity between thedivided block and the matching block corresponding to this divided blockis higher than that between the divided block and the unmatched block,so that the matching block corresponding to the divided block can bedetermined by searching for a matching block with the highest imagesimilarity with the divided block.

In the above embodiment, matching blocks are searched accurately foreach of the divided blocks of the second short-focus image based on theposition matching relationship or the comparison of the imagesimilarity.

In an embodiment, extracting the high-frequency component of thematching block for each of the divided blocks in the second short-focusimage can include: downsampling the long-focus image in which thematching block is located in the second long-focus image set such thatthe resolution of the downsampled image is the same as that of the firstshort-focus image, then performing interpolating on the downsampledimage such that the resolution of the interpolated image is equal tothat of the second short-focus image, and subtracting the correspondingblock of the matching block in the interpolated image from the matchingblock to obtain the high-frequency component of the matching block.

Wherein, subtracting the corresponding block of the matching block inthe interpolated image from the matching block can refer to subtracting,from the pixel value of the pixel for each pixel in the matching block,the pixel value of the corresponding point in the interpolated image ata position same as that of the pixel. Exemplary, suppose that the pixelvalue of the pixel whose pixel coordinates are (h, v) in the matchingblock is P1. Since the resolution of the interpolated image is equal tothe resolution of the second short-focus image, there must be acorresponding point whose pixel coordinates are also (h, v) in theinterpolated image. If the pixel value of this corresponding point isP2, then P1−P2 can be used as a new pixel value of the pixel whose pixelcoordinates are (h, v) in the matching block.

It can be understood that since the second long-focus image containsrelatively rich image details, the second long-focus image contains richhigh-frequency components, while the image information contained in thesecond short-focus image is all (or most) low-frequency components, soit can be considered that the matching block contains bothhigh-frequency components and low-frequency components, while thecorresponding block contains only low-frequency components (or containslow-frequency components and a small number of high-frequencycomponents). Therefore, the low-frequency information contained in thematching block can be removed by subtracting the corresponding block ofthe matching block in the interpolated image from the matching block, soas to extract the high-frequency component in the matching block.

Therefore, through this embodiment, the high-frequency component of thematching block is accurately extracted.

In an embodiment, in Step 201, the preset long-focus image setacquisition condition is met in a case including: when the binocularlong-focus and short-focus pan-tilt camera is started, or, when thedifference between the photometric value of the currently acquiredshort-focus image and that of the long-focus image in the last acquiredlong-focus image set is greater than a preset first threshold, or whenthe image similarity between divided blocks at same positions in thecurrently acquired short-focus image and in a long-focus image in thelast acquired long-focus image set is less than a preset secondthreshold.

It can be understood that when the difference between the photometricvalue of the currently acquired short-focus image and that of along-focus image in the last acquired long-focus image set is greaterthan the preset first threshold. it can be considered that the currentambient light changes compared with when the long-focus image set isacquired. When the image similarity between divided blocks at samepositions in the currently acquired short-focus image and in along-focus image in the last acquired long-focus image set is less thanthe preset second threshold, it can be considered that the currentmonitoring scenario changes compared with when the long-focus image setis acquired.

The captured images will be affected regardless whether the ambientlight changes or the monitoring scenario changes. Therefore, the imageinformation contained in the long-focus images in the long-focus imageset cannot be used as a reference for short-focus image reconstruction.That is, when the ambient light changes or the monitoring scenariochanges, the short-focus images cannot be accurately reconstructedaccording to the previously acquired long-focus image set, and thelong-focus image set needs to be re-acquired.

Therefore, in the above embodiment, when the binocular long-focus andshort-focus pan-tilt camera is started, or when the ambient lightchanges, or when the monitoring scenario changes, the long-focus imageset is re-acquired, thus ensuring the accuracy of the short-focus imagereconstruction.

In an embodiment, in Step 201, after the preset long-focus image setacquisition condition is met and before the pan-tilt platform of thebinocular long-focus and short-focus pan-tilt camera is driven to rotatethroughout the field-of-view range, the method can further include:

determining whether there is no alarm target in the currently acquiredshort-focus image and there is currently no image preview requirement;

when it is determined that there is no alarm target in the currentlyacquired short-focus image and there is currently no image previewrequirement, driving the pan-tilt platform of the binocular long-focusand short-focus pan-tilt camera to rotate throughout the field-of-viewrange, obtaining one long-focus image every preset horizontal and/orvertical angle, and constituting the first long-focus image set with allthe long-focus images acquired throughout the field-of-view range;

and, when it is determined that there is an alarm target in thecurrently acquired short-focus image or there is currently an imagepreview requirement, waiting for a preset time, and then returning tothe action of determining whether there is no alarm target in thecurrently acquired short-focus image and there is currently no imagepreview requirement.

In the above embodiment, it is started to acquire the long-focus imageset only when there is no alarm target in the currently acquiredshort-focus image and there is currently no image preview requirement,so as to avoid missing the alarm target and hindering the user's imagepreview requirement.

FIG. 3 shows the flow chart of an image reconstruction method providedin another embodiment of the present application. The specific steps areas follows.

Step 301: starting a binocular long-focus and short-focus pan-tiltcamera, wherein a pan-tilt platform rotates horizontally and verticallywithin its field-of-view range, and a long-focus camera and ashort-focus sensor acquire a short-focus image and a long-focus image inreal time.

Step 302: when the long-focus image set acquisition condition is met,driving the pan-tilt platform to rotate throughout the field-of-viewrange, and putting the long-focus image acquired by the pan-tiltplatform at each preset acquisition point into the first long-focusimage set until the pan-tilt platform has traversed all acquisitionpoints.

The long-focus image set acquisition condition is preset. The long-focusimage set acquisition condition is met in a case including: when thebinocular long-focus and short-focus pan-tilt camera is started, whenthe ambient light brightness changes, and/or when the scenario contentchanges (for example, the position of the binocular long-focus andshort-focus pan-tilt camera changes).

It can be determined whether the ambient light brightness changes bydetermining whether the difference between the photometric value of thecurrently acquired short-focus image and that of the long-focus image inthe last acquired long-focus image set is greater than the preset firstthreshold.

It can be determined whether the scenario content changes by determiningwhether the image similarity between the currently acquired short-focusimage and the divided block at a position same as that of a long-focusimage in the last acquired long-focus image set is less than the presetsecond threshold.

If the initial position of the pan-tilt platform is zero, and thehorizontal and vertical azimuth angles in this case are both 0, thepreset acquisition points can be set as follows: one acquisition pointat every 25 degrees horizontally and one acquisition point at every 12degrees vertically from 0, or, one acquisition point at every 12.5degrees horizontally and one acquisition point at every 6 degreesvertically from 0, other degrees are also possible, which is not limitedin the present application.

Step 303: when receiving a zoom request with a zoom factor from anexternal input, performing interpolating on the current short-focusimage (set as the first short-focus image) by using a presetinterpolation algorithm to obtain the second short-focus image thatmeets the zoom request.

The interpolation algorithm is mature. The used algorithm is not limitedin this embodiment, for example, bilinear interpolation, bicubicinterpolation, or minimum curvature interpolation.

Step 304: calculating the reduction factor corresponding to thelong-focus image when a target in the long-focus image in the firstlong-focus image set acquired in Step 302 and the same target in thesecond short-focus image are matched in size, based on the focal lengthsof the long-focus lens and the short-focus lens, and the zoom factorused in Step 303, and using the reduction factor to downsample eachlong-focus image in the last acquired first long-focus image set toobtain the second long-focus image set.

Step 305: dividing the second short-focus image into multiplepreset-size non-overlapping divided blocks, and searching, for each ofthe divided blocks, all long-focus images in the second long-focus imageset for a block (i.e., a best matching block) that matches best withthis divided block, and recording the similarity between each of thedivided blocks and the matching block for this divided block.

Specifically, in this step, before searching for a matching block foreach of the divided blocks, a long-focus image that overlaps infield-of-view with the second short-focus image is firstly selected inthe second long-focus image set based on the angle of the pan-tiltplatform when the first short-focus image is acquired and the angle ofthe pan-tilt platform when each long-focus image in the first long-focusimage set is acquired. Then, for each of the divided blocks, the bestmatching block is searched for in the selected long-focus image.Wherein, the best matching block can refer to either a matching blockwith a matched position or a matching block with the highest imagesimilarity. The matched position and image similarity have beendescribed in the previous description and are not discussed here.

For example, if the position of the pan-tilt platform is (x, y) when thefirst short-focus image is acquired, x being the horizontal angle and ybeing the vertical angle, then the field-of-view covered by the firstshort-focus image is shown in FIG. 4 . The long-focus image in thisrange of the field-of-view can be selected from the first long-focusimage set.

In this step, one of the following methods can be used to search alllong-focus images in the second long-focus image set for the bestmatching block.

1. Search, for each of the divided blocks of the second short-focusimage, the selected long-focus image for a matching block that matchesthe divided block in position based on the angle of the pan-tiltplatform when the first short-focus image is acquired, the angle of thepan-tilt platform when each long-focus image in the first long-focusimage set is acquired and the field-of-view difference between theshort-focus lens and the long-focus lens;

Matching in position means that the divided block and the matching blockcorrespond to the same position in the monitoring scenario.

2. Search, for each of the divided blocks, the selected long-focus imagefor a matching block with the highest image similarity with the dividedblock by using a preset template matching algorithm.

Template matching algorithms can be MAD (Mean Absolute Difference), SAD(Sum of Absolute Difference), or SSD (Sum of Squared Difference), etc.

Step 306: when matching blocks are found for all divided blocks of thesecond short-focus image, downsampling the long-focus image in whicheach matching block is located so that the resolution of the downsampledimage is equal to the resolution of the first short-focus image; andthen performing interpolating on the downsampled image by using a presetinterpolation algorithm so that the resolution of the interpolated imageis equal to the resolution of the second short-focus image, theinterpolated image being called a low-frequency long-focus image;subtracting, for each matching block, the corresponding block of thematching block in the low-frequency long-focus image from the matchingblock to obtain the high-frequency component of the matching block.

Step 307: calculating the fusion weight for each matching block

${W_{k} = ( \frac{E_{k}}{E_{\min}} )^{- \alpha}},$

wherein W_(K) is the fusion weight of the matching block for the kthdivided block of the second short-focus image,

${E_{k} = \frac{1}{S_{k}}},$

S_(K) is the similarity between the kth divided block of the secondshort-focus image and the matching block,

${E_{\min} = \frac{1}{S_{\max}}},$

S_(max) is the maximum similarity among the similarities between alldivided blocks of the second short-focus image and the matching blocksfor the divided blocks, and α is a preset value. Based on experience,generally, α can be 0.5, or values other than 0.5, such as 0.3, 1.2, 3,etc.

Step 308: for each of the divided blocks in the second short-focusimage, multiplying the high-frequency component of the matching block ofthe divided block by the corresponding fusion weight and thensuperimposed it with the divided block to obtain the reconstructed blockof the divided block. All the reconstructed blocks of the divided blocksmake up the reconstructed short-focus image.

After obtaining the reconstructed short-focus image in Step 205 and Step308, the reconstructed short-focus image can be fused with the currentlyacquired long-focus image to obtain a fused image finally output to theuser. The fusion process is specifically as follows.

Suppose that the currently acquired long-focus image is a thirdlong-focus image, then:

Step 01: calculating a reduction factor corresponding to the thirdlong-focus image when a target in the third long-focus image and thesame target in the reconstructed short-focus image are matched in size;

Step 02: performing a reduction process on the third long-focus imageaccording to the calculated reduction factor, to obtain a fourthlong-focus image;

Step 03: calculating the position of the fourth long-focus image in thereconstructed short-focus image when a target in the fourth long-focusimage and the same target in the reconstructed short-focus image arematched in position according to the relative angle between the currentlong-focus lens and the short-focus lens;

Step 04: superimposing the fourth long-focus image on the reconstructedshort-focus image according to the position of the fourth long-focusimage in the reconstructed short-focus image, to obtain the fused imagefinally output to the user.

It can be seen that since the original short-focus image is processed bysuper-resolution reconstruction in the embodiment of the presentapplication, the difference in definition between the picture in themiddle and the picture around of the fused image finally output to theuser is greatly reduced.

FIG. 5 is a schematic structural diagram of an image reconstructiondevice provided in an embodiment of the present application, whichmainly includes a memory 501 and a processor 502 that can access thememory 501 having stored instructions therein which, when executed bythe processor 502, cause the processor 502 to implement the steps, suchas steps 201-205, or steps 301-308, of the method.

The present application also provides a binocular long-focus andshort-focus pan-tilt camera which includes a short-focus camera, along-focus camera and a pan-tilt platform, wherein the relativepositions of the short-focus camera and the long-focus camera are fixed,and the short-focus camera and the long-focus camera rotatesimultaneously with the pan-tilt platform. Further, each of theshort-focus camera and the long-focus camera has an interface tocommunicate with the image reconstruction device, and the short-focuscamera and the long-focus camera send the original short-focus image andthe original long-focus image respectively acquired by the short-focuscamera and the long-focus camera to the image reconstruction devicethrough the respective communication interfaces.

In another embodiment of the present application, there is provided acomputer readable storage medium having stored therein a computerprogram which, when executed by a processor, causes the processor toperform steps of any image reconstruction method described above.

In still another embodiment of the present invention, there is alsoprovided a computer program product containing instructions that, whenrunning on a computer, cause the computer to perform the imagereconstruction method in any one of the embodiments described above.

The description is only for preferred embodiments of the presentapplication, and is not intended to limit the present application. Anymodifications, substitutions, improvements, etc., which are made withinthe spirit and principles of the present application, shall fall withinthe protection scope of the present application.

1. An image reconstruction method, comprising: when a preset long-focusimage set acquisition condition is met, driving a pan-tilt platform of abinocular long-focus and short-focus pan-tilt camera to rotatethroughout a field-of-view range, obtaining one long-focus image everypreset horizontal and/or vertical angle, and constituting a firstlong-focus image set with all long-focus images acquired throughout thefield-of-view range, wherein adjacent long-focus images overlappartially; receiving a zoom request, and performing interpolating on afirst short-focus image currently acquired by the binocular long-focusand short-focus pan-tilt camera to obtain a second short-focus imagethat meets the zoom request; downsampling each long-focus image in thefirst long-focus image set to obtain a second long-focus image set,wherein a target in each long-focus image in the second long-focus imageset and the same target in the second short-focus image are matched insize; dividing the second short-focus image into blocks, and searching,for each of the divided blocks, all long-focus images in the secondlong-focus image set for a matching block; fusing each of the dividedblocks in the second short-focus image with the matching block for thisdivided block to obtain a reconstructed short-focus image; wherein thebinocular long-focus and short-focus pan-tilt camera is composed of ashort-focus camera and a long-focus camera, relative positions of theshort-focus camera and the long-focus camera are fixed, and theshort-focus camera and the long-focus camera rotate simultaneously withthe pan-tilt platform.
 2. The method of claim 1, wherein, fusing each ofthe divided blocks in the second short-focus image with the matchingblock for this divided block comprises: extracting a high-frequencycomponent of the matching block for each of the divided blocks, andfusing each of the divided blocks with the high-frequency component ofthe matching block for this divided block.
 3. The method of claim 2,wherein, fusing each of the divided blocks in the second short-focusimage with the high-frequency component of the matching block for thisdivided block comprises: assigning a fusion weight for each matchingblock according to a principle that the larger a similarity between thematching block and its corresponding divided block is, the larger thefusion weight of the matching block is; multiplying the high-frequencycomponent of each matching block by the corresponding fusion weight toobtain a result and then superimposing the result with the correspondingdivided block, to obtain the reconstructed short-focus image.
 4. Themethod of claim 3, wherein, assigning the fusion weight for eachmatching block according to the principle that the larger a similaritybetween the matching block and its corresponding divided block is, thelarger the fusion weight of the matching block is comprises: calculating${W_{k} = ( \frac{E_{k}}{E_{\min}} )^{- \alpha}},$ whereinW_(K) is a fusion weight of a matching block for a kth divided block ofthe second short-focus image, ${E_{k} = \frac{1}{S_{k}}},$  S_(K) is asimilarity between the kth divided block of the second short-focus imageand the matching block, ${E_{\min} = \frac{1}{S_{\max}}},$  S_(max) is amaximum similarity among similarities between all divided blocks of thesecond short-focus image and the matching blocks for the divided blocks,and α is a preset value.
 5. The method of claim 1, wherein, searching,for each of the divided blocks, all long-focus images in the secondlong-focus image set for the matching block comprises: searching, foreach of the divided blocks of the second short-focus image, all thelong-focus images in the second long-focus image set for a matchingblock that matches the divided block in position according to an angleof the pan-tilt platform when the first short-focus image is acquired,an angle of the pan-tilt platform when each long-focus image in thefirst long-focus image set is acquired and a field-of-view differencebetween a short-focus lens and a long-focus lens; or searching, for eachof the divided blocks, all the long-focus images in the secondlong-focus image set for a matching block having a highest imagesimilarity with the divided block by using a preset template matchingalgorithm.
 6. The method of claim 2, wherein, extracting thehigh-frequency component of the matching block for each of the dividedblocks in the second short-focus image comprises: downsampling thelong-focus image in which the matching block is located in the secondlong-focus image set such that a resolution of the downsampled image isthe same as that of the first short-focus image, then performinginterpolating on the downsampled image such that a resolution of theinterpolated image is equal to that of the second short-focus image, andsubtracting a corresponding block of the matching block in theinterpolated image from the matching block to obtain the high-frequencycomponent of the matching block.
 7. The method of claim 1, wherein, thepreset long-focus image set acquisition condition is met in a casecomprising: when the binocular long-focus and short-focus pan-tiltcamera is started, or, when a difference between a photometric value ofa currently acquired short-focus image and a photometric value of along-focus image in a last acquired long-focus image set is greater thana preset first threshold, or when an image similarity between dividedblocks at same positions in the currently acquired short-focus image andin a long-focus image in the last acquired long-focus image set is lessthan a preset second threshold.
 8. The method of claim 1, wherein, afterthe preset long-focus image set acquisition condition is met and beforethe pan-tilt platform of the binocular long-focus and short-focuspan-tilt camera is driven to rotate throughout the field-of-view range,the method further comprises: determining whether there is no alarmtarget in a currently acquired short-focus image and there is currentlyno image preview requirement, when it is determined that there is noalarm target in the currently acquired short-focus image and there iscurrently no image preview requirement, driving the pan-tilt platform ofthe binocular long-focus and short-focus pan-tilt camera to rotatethroughout the field-of-view range, obtaining one long-focus image everypreset horizontal and/or vertical angle, and constituting the firstlong-focus image set with all long-focus images acquired throughout thefield-of-view range; and when it is determined that there is an alarmtarget in the currently acquired short-focus image or there is currentlyan image preview requirement, waiting for a preset time, and thenreturning to the action of determining whether there is no alarm targetin the currently acquired short-focus image and there is currently noimage preview requirement.
 9. An image reconstruction device, whichcomprises a memory and a processor that can access the memory havingstored therein instructions which, when executed by the processor, causethe processor to implement steps of the method of claim
 1. 10. Abinocular long-focus and short-focus pan-tilt camera, which comprises ashort-focus camera, a long-focus camera and a pan-tilt platform, whereinrelative positions of the short-focus camera and the long-focus cameraare fixed, and the short-focus camera and the long-focus camera rotatesimultaneously with the pan-tilt platform; and each of the short-focuscamera and the long-focus camera has an interface to communicate withthe image reconstruction device of claim 9, and the short-focus cameraand the long-focus camera send an original short-focus image and anoriginal long-focus image respectively acquired by the short-focuscamera and the long-focus camera to the image reconstruction devicethrough the respective communication interfaces.
 11. A non-transitorycomputer readable storage medium having stored therein a computerprogram which, when executed by a processor, causes the processor toperform steps of the method of claim 1.